Accurate background model is the paramount base for object extracting and tracing. In response to swing objects which part quasi-periodically changed in intricate scene, based on multi-Gaussian background model, a new Quasi-Periodic Background Algorithm (QPBA) was proposed to suppress the swing objects and establish an accurate and stable background model. The specific process included: According to multi-Gaussian background model, the object classification in scene was set up, and the effect on Gaussian model's parameters caused by swing objects was analyzed. By using color distribution values as samples to establish Gaussian model to keep swing pixels, the swing model in swing pixels was integrated into background model with weight factors of occurrence frequency and time interval. Comparison among QPBA and the classical background modeling algorithms such as GMM (Gaussian Mixture Model), ViBe (Visual Background extractor) and CodeBook was put forward, and the results were assessed in aspects of quality, quantity and efficiency. It shows that QPBA has a more obvious suppression on swing objects, and its fall-out ratio is less than 1%, so that it can handle the scene with swing objects. At the same time, its correct detection number is consistent with other algorithms, thus the moving objects can be reserved perfectly. In addition, the efficiency of QPBA is high, and its resolving time is approximate to CodeBook, which can satisfy the requirements of real-time computation.